Robust Conditional Probabilities

Conditional probabilities are a core concept in machine learning. For ex-ample, optimal prediction of a label Y given an inputX corresponds to maximizingthe conditional probability of Y given X . A common approach to inference tasksis learning a model of conditional probabilities. However, these models are oftenbased on strong assumptions (e.g., log-linear models), and hence their estimate ofconditional probabilities is not robust and is highly dependent on the validity oftheir assumptions.Here we propose a framework for reasoning about conditional probabilities withoutassuming anything about the underlying distributions, except knowledge of theirsecond order marginals, which can be estimated from data. We show how thissetting leads to guaranteed bounds on conditional probabilities, which can be calcu-lated efficiently in a variety of settings, including structured-prediction. Finally, weapply them to semi-supervised deep learning, obtaining results competitive withvariational autoencoders.